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ABSTRACT 

An evaluation was conducted of a summer institute on 
assessment in adult literacy held at the University of Pittsburgh. 
The summer institute was designed to enable program administrators 
and instructors to hear experts in the field of adult assessment 
discuss, in depth, the issues that are common to all adult literacy 
providers. Participants had opportunities to broaden their knowledge 
base about formal and informal assessment and to participate in 
problem-solving activities about assessment issues relevant to their 
day-to-day professional lives. Each participant received institute 
evaluation forms for the small group sessions, workshops, and overall 
effectiveness of the institute, including quality of the keynote 
addresses. The evaluation found that of the 95 participants, 43 
percent were administrators, 37 percent were instructors, 12 percent 
were counselors, and 8 percent were volunteers. Fifty percent of the 
participants identified themselves as experienced in the area of 
adult literacy. The majority of participants agreed that the content 
of the workshops and small group sessions was relevant to their 
needs, well organized, and highly effective, with an adequate balance 
of information-giving, hands-on activities, and time to ask questions 
and interact. The study concluded that, overall, the institute was 
successful and accomplished its goals in addressing the needs of a 
diverse group of literacy providers. Recommendations to improve 
future workshops included the following: (1) keynote speakers need to 
incorporate adequate opportunities for gro'ip interaction; (2) keynote 
speakers need to provide practical examples so participants can make 
connections between theory and practice; (3) participants would 
benefit from more handouts so that they could review the materials; 
and (4) participants should be provided with lists of references. 
(The questionnaires used in the evaluation are included in the 
report.) (KC) 
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Summer Institute: Focus on Assessment in Adult Literacy Education 



Introduction 

An issue of importance across adult literacy programs is assessment and 
diagnosis. Adult literacy educators are concerned about the adequacy of measures 
used to determine learning gain and program impact. Moreover, there is much 
interest in alternative assessment and its potential as an interactive tool that 
informs instructional decision-making. 

There are many unanswered questions in the field of adult literacy regarding 
how to choose appropriate test measures, how to use the information these 
measures yield, how to measure the actual impact of instruction, and how to 
develop and use informal assessment so that students, instructors, and program 
administrators have more comprehensive information about student learning and 
progress. This institute provided participants with a three-day comprehensive look 
at all these assessment issues in both a general way and in ways specific to the needs 
generated by particular types of programs. Participants had the opportunity to fill in 
knowledge gaps about assessment by hearing from nationally recognized leaders in 
the field, questioning them, and applying what they had learned under the guidance 
of knowledgeable professionals in the field of adult literacy. 
Goals and Objectives 

The overall goal of the institute was to enable program administrators and 
instructors to hear experts in the field of adult assessment discuss, in depth, the 
issues which art common to all adult literacy providers. All participants, regardless 
of experiential level, had opportunities to broaden their knowledge base 
about formal and informal assessment and participate in problem solving activities 
about assessment issues relevant to their day-to-day professional lives. 
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The objectives of the institute were to provide program administrators, 
novice instructors, and experienced instructors with opportunities to: 

(1) Learn more about standardized and informal assessment measures, 
what they have to offer and their limitations; 

(2) Learn more about specific instruments that are available for adult 
literacy providers to assess reading and writing and to inform 
instruction: 

(3) Learn more about how to evaluate whether an instructional program 
is achieving its goals; and 

(4) Interact with other adult literacy educators about assessment issues and 
tools. 

Each of the objectives was addressed in three ways throughout the institute. 
First, all participants heard nationally known experts in the field of testing and 
assessment and began to develop a general knowledge base of standardized testing, 
informal assessment, and use of assessment to inform instruction. Second, 
participants engaged in small group experiences which enabled them to focus on 
areas of particular interest to them and further expand upon information received 
from keynote speakers. Finally, participants participated in workshop sessions to 
apply the knowledge they had acquired by becoming actively involved with 
colleagues in addressing issues of interest to them and finding solutions. Therefore, 
the institute's goals and objectives were met by beginning with the broad issues and 
providing opportunities for participants to address these issues as they relate to their 
specific program needs. 
Procedures 

The institute was designed to progress from the general to the specific. 
Therefore, each day began with a keynote address of general interest to all 
participants. After the keynote, participants chose from five small group sessions 



for which they had preregistered. On day two of the institute, participants 
preregistered for both a small group session and a workshop session, (see Appendix 
A). 

Institute activities involved participants in a variety of learning situations. 
All participants had the opportunity to attend information giving sessions, small 
group interactive sessions, and sessions focusing on authentic learning tasks. 

Keynote addresses used a lecture format to convey information to the large 
group. Time was allotted for questions, so participants had some opportunity to 
address specific program needs. 

Small group sessions combined information giving with group intaraction. 
Many of these sessions involved people in problem-solving and discussion 
activities. Small group presenters brought hand-outs with information that could 
be used upon returning to participants' classrooms. 

Workshop sessions offered minimal information giving and specifically 
focused on hands-on application. Each workshop presented a different type of 
assessment instrument, gave participants a chance to see it demonstrated, and 
enabled participants to engage in an authentic learning task using the instrument. 
Workshop presenters provided participants with numerous hand-outs such as 
copies of assessment instruments and instructional ideas to use in their own 
classrooms. 

Keynote sessions were attended by all 95 participants. Small group session 
and workshop sessions were limited to 25 participants per session. 
Evaluation Approach 

At registration, each participant received an institute folder containing 
evaluation forms for the small group sessions, workshops, and overall effectiveness 
of the institute including the quality of the keynote addresses. (See Appendix B) 
The evaluation forms were completed and collected by volunteers from a local 

3 

5 



literacy agency in an ongoing manner throughout the course of the institute. The 
information from these evaluations was compiled and the results summarized. 
Evaluation Results 

Analysis of results from the evaluation forms yielded information about: (1) 
the content, structure, and personal benefits of the small group sessions and 
workshops; (2) the quality of the keynote addresses; and (3) the overall quality and 
organization of the institute. Results are discussed in general terms and are 
combined by session type e.g. small group, workshop, keynote . Data from all small 
group sessions, workshops, keynote addresses, and overall effectiveness of the 
institute were also aggregated to assess overall impact, (see Table 1- 4) 

Participants 

Those who attended the workshop were generally a knowledgeable group of 
professionals who identified themselves as experienced (50%) in the area of adult 
assessment. Of the 95 participants, 43% were administrators, 37% were instructors, 
12% were counselors, and 8% were volunteers. 

Small Group Sessions. 

Of the ten small group sessions conducted, the majority of participants agreed 
that the content of the sessions was relevant to their needs and well organized. 
Similarly, the majority felt strongly that there was adequate opportunities for group 
participation and interaction and sharing among the participants. In all small group 
sessions, participants overall indicated they left with new skills and techniques 
which would help them be more effective in their current position. 

Evaluation results for the ten small group sessions indicated that the 
meetings were well received and highly effective. Specifically, 69% of the 
participants felt the session content was relevant to their needs and 84% believed 
the presentations were well organized. Sixty six percent felt the structure of the 
session allocated sufficient time for the presentations while 95% found the time 
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given for hands-on experiences was excellent, and 73% were satisfied with the time 
allotted for group interaction. Over 50% of the participants agreed that they had 
learned new skills and techniques from the presenters and that they had gained 
knowledge that would be useful in their current position. Sixty eight percent said 
that their interaction with the presenter was beneficial. 
Workshops. 

Like the small group sessions, the five workshops were received well. 
Overall, the participants agreed that the content was what they expected, relevant to 
their needs, and well organized. Participants were pleased with the pacing of the 
information presented. They seemed to feel that there was an adequate balance 
between information giving, hands-on activities, and time to ask questions and 
interact. For four of the five workshops, participants felt that they had acquired 
useful new skills and techniques that would help them in their current position. 
When asked how the workshops could improve, the majority of participants agreed 
that it would be helpful to have a break during a three hour workshop. 

Overall evaluation results of the five workshops indicated that the 
workshops, like the small group sessions, were well received and useful. 
Specifically, in terms of content, 71% of the participants felt that the workshops met 
their needs and 84% found them to be well organized. Over 75% of participants 
responded positively to workshop structure. They believed that there was sufficient 
time allotted for presentation of material, for participation in activities, and for 
interaction and sharing among participants. Over 60% of participants felt they 
acquired new skills which they expected to use and new knowledge which would 
help them in their positions. Sixty six percent of participants felt interaction with 
workshop presenters was beneficial. 



Keynote Addresses 

For the three keynote addresses, the majority of participants agreed that the 
overall content and delivery of the presentations was excellent. Specifically, 
participants felt that keynoters provided them with new, useful information and 
interesting perspectives on adult reading assessment. They believed that the 
handouts would be helpful to them in their current positions. Participants overall 
felt that the personal interaction with the keynote speakers was beneficial. 

Overall Effectiveness of the Institute 

Overall, evaluation results of the institute were positive, indicating goals 
were met. Ninety two percent of those completing evaluation forms rated the 
organization of the institute as excellent or good. Ninety two percent felt the 
content of the sessions was excellent or good; 89% found the structure of the 
sessions to be excellent or good; and 90% felt that the institute was beneficial to adult 
educators. 

Recommendations 

Comments from keynote sessions, small group sessions, and workshops were 
generally laudatory and it was obvious that participants were actively involved in 
all facets of the Institute. Evidence of their interest in the topic came across in 
multiple comments requesting more time for all types of sessions. Assessment is a 
particularly complex issue and one hour sessions and even three hour workshops 
did not provide some participants with the indepth information they sought. 

The purpose of the institute was to provide a broad based menu of options in 
the area of assessment and this was accomplished. Because of its general nature, 
however, it was not possible for participants to obtain an indepth understanding of 
any one aspect of assessment. 

Some of the following recommendations may be helpful in developing 
future workshops. First, keynote speakers need to be encouraged to incorporate 
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adequate opportunities for group interaction so participants can ask questions and 
clarify issues. Second, keynote speakers need to be reminded to provide practical 
examples so participants can make the connections between theory and practice. 
Third, across all types of sessions, participants would benefit from more handouts so 
they could have the opportunity to walk away from presentations with something 
they could revisit at their leisure and/or share with colleagues. Finally, participants 
should be provided with lists of references so they could become more 
knowledgeable about where to find additional information to answer current and 
future questions. 

Overall, this Institute was successful and accomplished its goals in addressing 
the needs of a diverse group of literacy providers. Participants were given a unique 
opportunity to find answers to complex assessment issues from among the most 
knowledgeable people in the field. As a result, literacy providers across the state 
should be using standardized measures with more understanding of the 
information they can and cannot yield, feel more confident about integrating other 
forms of assessment into their programs, and be more adept at using results of 
testing to inform instruction. 
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Table 1. 



Summary of Small Group Sessions 



Smali Groups 



Content! 

1. Relevant to my needs 

2. Well organized 



Yes 
72% 
88% 



To A Degree 

27% 
12% 



No 

1% 
0% 



Structure: Excellent 

3. Time allocated for 

presentation was 64% 

4. Time allocated for participation 

in activities was 70% 

5. Time for interaction and sharing 
among participants was 72% 



fair 

35% 
29% 
26% 



1% 



1% 



2% 



Results: 



Yes 



6. I have new skills and techniques 

I expect to use 60% 

7. I gained knowledge that will 

help me in my position 69% 

8. I benefited from interaction 

with the presenter 73% 



To A Degree 

37% 
29% 
24% 



3% 



2% 



3% 
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Table 2. 



Summary of Workshop Sessions 



Workshops 

Content: Yes To A Deg ree 

1. Relevant to my needs 71% 25% 

2. Well organized 84% 16% 

Structure : Excellent Eak 



3. Time allocated for 
presentation was 

4. Time allocated for participation 
in activities was 

5. Time for interaction and sharing 
among participants was 



Results: 

6. I have new skills and techniques 
I expect to use 

7. I gained knowledge that will 
help me in my position 

8. I benefited from interaction 
with the presenter 



72% 23% 

72% 23% 

80% 18% 

lea To A Deg ree 

60% 36% 

64% 33% 

66% 33% 



Table 3. Summary of Responses to Keynote Addresses 





Excellent 


Good 


Average 


Poor 


Standardized Testing 
Page Bristow 


46% 


41% 


11% 


1% 


Alternative Assessment 
Susan Lytle 


42% 


39% 


17% 


1% 


Assessing Instruction to 
Inform 
Nancy Padak 


72% 


23% 


5% 


0% 
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Table 4. 



Summary of Responses to Overall Institute 





Excellent 


Good 


Average 


Poor 


Organization of 
the Institute 


45% 


47% 


8% 


0% 


Content of the 










Sessions 


45% 


44% 


11% 


0% 


Structure of the 










Sessions 


32% 


56% 


11% 


0% 


Overall Benefit to 










Me as an Adult 
Literacy Educator 


41% 


46% 


12% 


1% 
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Summer InStitllfff; Focus on ^sgfiSamtni Evaluation Vnrrn 

SMALL GROUP SESSION - DAY 1 

Please fill out one evaluation form for each small group session that you attend. 



Small Group Title: 



Presenter(s): 



What is your level of expertise in the area of adult literacy! 
Novice Experienced Administrator Institute Volunteer 



REACTION (Please check one response in each line) 

Content : Yes To a Degree No 

1. Relevant to my needs 



2. Well organized 
Structure: 

3. Time allocated for 
presentation was 

4. Time allocated for participation 
in activities was 

5. Time for interaction and sharing 
among participants was 

Results : 

6. I have new skills and techniques 
I expect to use 

7. I gained knowledge that will 
help me in my position 

8. I benefitted from interaction 
with the presenter 



Excellent 



Yes 



Fair 



Poor 



To a Degree No 



How would you improve this session? 



Additional Comments: 
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Summer TnsH*nfp» 



■Focus on Ass^^ut Ev»in«Hnn Fftrm 



SMALL GROUP SESSION - DAY 2 

JW out one evaluation form Jor ; each small gttfc session that you attend, f 



Novice 



Experienced 



Small Group Title: 

What is your level of expertise in the area" of ^ifi/r //^cy? 



Administrator ___ Institute Volunteer 



■4 

REACTION (Please check one response in each line) 

Yes To a Degree No 



Content : 

1. Relevant to my needs 



2. Well organized 
Structure: 

3. Time allocated for 
presentation was 

4. Time allocated for participation 
in activities was 

5. Time for interaction and sharing 
among participants was 

Results : - 

6. I have new skills and techniques 
I expect to use 

7. I gained knowledge that will 
help me in my position 

8. I benefitted from interaction' j^fe^ 
with the presenter :.rv ■■ - .■ : K 

■■ . m$> : ] 

How would you improve this session? A 



Additional Comments: 



Excellent 



Fair 



Poor 



Yes 



To a Degree No 
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Summer Tnsfiflltf; Focus on A^o«m>nt Evaluation Form 

WORKSHOP - DAY 2 

Workshop Title: ^ Presenter(s): 



What is your level of expertise in the area of adult literacy! 
Novice Experienced Administrator Institute Volunteer 

REACTION (Please check one response in each line) 

c °Ptent : Yes To a Degree No 

1. Relevant to my needs 



2. Well organized 



gtructurg: Excellent Fair Poor 

3. Time allocated for 
presentation was 



4. Time allocated for participation 
in activities was 

5. Time for interaction and sharing 
among participants was 

Results : 

6* I have new skills and techniques 
I expect to use 

7. I gained knowledge that will 
help me in my position 

8. I benefitted from interaction 
with the presenter 



How would you improve this workshop? 



Yes To a Degree No 



Additional Comments: 
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Summer Tnsfifnff; — Focus on Aasgssmsni F vaiiiatinn Form 

SMALL GROUP SESSION - DAY 3 
Please fill out one evaluation form for each small group session that you attend. 



Small Group Title: 



Presenter(s): 



What is your level of expertise in the area of adult literacy! 
Novice Experienced Administrator Institute Volunteer 



REACTION (Please check one response in each line) 

£ea*SBfc Yes To a Degree No 

1. Relevant to my needs 



2. Well organized 
Structure: 

3. Time allocated for 
presentation was 

4. Time allocated for participation 
in activities was 

5. Time for interaction and sharing 
among participants was 

Results : 

6. I have new skills and techniques 
I expect to use 

7. I gained knowledge that will 
help me in my position 

8. I benefitted from interaction 
with the presenter 



Excellent 



Fair 



Poor 



Yes 



To a Degree No 



How would you improve this session? 



Additional Comments: 
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SUMMER INSTITUTE: FOCUS ON ASSESSMENT 
EVALUATION FORM 

Please help us plan for future institutes by completing the following: 

General Information 

Please indicate the following: 

Days attended: Mon. , Tues. , Wed. . 

What is your level of expertise in the area of adult literacy? 

Novice Experienced Administrator Institute Volunteer 

Keynote Speakers 

(Please rate each speaker whose session you attended. Please circle the appropriate number.) 
Page Bristnw 

4 Excellent Presentation 3 Good 2 Average 1 Poor 0 Didn't Attend 
Comments: 



Susan Lytle 

4 Excellent Presentation 3 Good 2 Average 1 Poor 0 Didn't Attend 
Comments: 



Nancy Padak 

4 Excellent Presentation 3 Good 2 Average 1 Poor 0 Didn't Attend 
Comments: 



Small Group and Workshop Sessions 

Please list the presenters of each session that you attended and rate each as 
(4) Excellent (3) Good (2) Average (1) Poor 



1. Presenter: 

2. Presenter: 

3. Presenter: 
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Rating: 
Rating: 
Rating: 



Panel Discussion 
Program Evaluation: 

4 Excellent 3 Good 2 Average 1 Poor 

Overall Institute 
Organizati on of the Institute? 

4 Excellent Presentation 3 Good 2 Average 1 Poor 0 Didn't Attend 
Comments: 



Content of Sessions: 




4 Excellent Presentation 3 Good 2 Average 


1 Poor 0 Didn't Attend 


Comments: 




Structure of the Sessions; 




4 Excellent Presentation 3 Good 2 Average 


1 Poor 0 Didn't Attend 


Comments: 





Overall benefit to me as an Adult Literacy E ducator- 

4 Excellent Presentation 3 Good 2 Avenge 1 Poor 0 Didn't Attend 



Comments: 



Recommendations (Include suggestions for speakers and topics.) 



ERIC 



20 



(NOTE: You may adapt space as needed for the sections. However, 
limit the abstract to one page.) 

ABSTRACT PAGE 

Title: Summe r Institute: Focus on Assessment In Adult Literacy Education 
Project No . : 99-4003 Funding: $49.990 

Project Director: Rita M. Bean Phona No. : (412) 648-1774 

Contact Person: Michael Crouch Phone No. : (412) 624-7400 

Arencv Addres 5 University of Pittsburgh, School of Education, 5T01 Forbes 
Quadrangle, Pittsburgh, PA 15260 

Purge Sg : 

The overall goal of the institute was to enable program administrators and instructors to hear 
experts in the field of adult assessment discuss, in depth, the issues which are common to all adult 
literacy providers. Participants had opportunities to broaden their knowledge base about formal and 
informal assessment and participate in problem solving activities about assessment issues relevant to 
their day-to-day professional lives. 

On each day all participants had the opportunity to attend information giving sessions, small 
group interactive sessions, and sessions focusing on authentic learning tasks. Each oiticiDant 
rrceived institute evaluation forms for the small group sessions, workshops, and overall effectiveness 
of the institute including the quality of the keynote addresses. 
Summary Qff Findings : 

Of the 95 participants, 43% were administrators, 37% instructors, 12% counselors, and 8% 
were volunteers. Fifty percent of the participants identified themselves as experienced in the area of 
adult literacy. The majority of participants agreed that the content of the workshops and small group 
sessionswas relevant to their needs, well organized, and highly effective, with an adequate balance 
between information giving, hands-on activities, and time to ask questions and interact 
Comments, (Conclusions/ F^ndincrs, Barriers, if anv) : 

Overall, this institute was successful and accomplished its goals in addressing the needs of a 
diverse group of literacy providers. The following recommendations may be helpfiil in developing 
future workshops. First, keynote speakers need to incorporate adequate opportunities for group 
interaction. Second, keynote speakers need to provide practical examples so participants can make * 
connections between theory and practice. Third, participants would benefit from more handouts so 
they could have the opportunity revisit information at their leisure and/or share with colleagues 
Finally, participants should be provided with lists of references so they could become more 
knowledgeable about where to find additional information to answer current and future questions. 

Products (if applicable) : 
Paper 

Descriptor?. (To be completed only by Bureau staff) : 
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